A Survey on Deep Learning in Medical Image Registration: New Technologies, Uncertainty, Evaluation Metrics, and Beyond

Over the past decade, deep learning technologies have greatly advanced the field of medical image registration. The initial developments, such as ResNet-based and U-Net-based networks, laid the groundwork for deep learning-driven image registration. Subsequent progress has been made in various aspects of deep learning-based registration, including similarity measures, deformation regularizations, and uncertainty estimation. These advancements have not only enriched the field of deformable image registration but have also facilitated its application in a wide range of tasks, including atlas construction, multi-atlas segmentation, motion estimation, and 2D-3D registration. In this paper, we present a comprehensive overview of the most recent advancements in deep learning-based image registration. We begin with a concise introduction to the core concepts of deep learning-based image registration. Then, we delve into innovative network architectures, loss functions specific to registration, and methods for estimating registration uncertainty. Additionally, this paper explores appropriate evaluation metrics for assessing the performance of deep learning models in registration tasks. Finally, we highlight the practical applications of these novel techniques in medical imaging and discuss the future prospects of deep learning-based image registration

DRIMET: Deep Registration for 3D Incompressible Motion Estimation in Tagged-MRI with Application to the Tongue

Tagged magnetic resonance imaging (MRI) has been used for decades to observe and quantify the detailed motion of deforming tissue. However, this technique faces several challenges such as tag fading, large motion, long computation times, and difficulties in obtaining diffeomorphic incompressible flow fields. To address these issues, this paper presents a novel unsupervised phase-based 3D motion estimation technique for tagged MRI. We introduce two key innovations. First, we apply a sinusoidal transformation to the harmonic phase input, which enables end-to-end training and avoids the need for phase interpolation. Second, we propose a Jacobian determinant-based learning objective to encourage incompressible flow fields for deforming biological tissues. Our method efficiently estimates 3D motion fields that are accurate, dense, and approximately diffeomorphic and incompressible. The efficacy of the method is assessed using human tongue motion during speech, and includes both healthy controls and patients that have undergone glossectomy. We show that the method outperforms existing approaches, and also exhibits improvements in speed, robustness to tag fading, and large tongue motion.Comment: Accepted to MIDL 2023 (full paper

Validation of a Robust Method for Quantification of Three-Dimensional Growth of the Thoracic Aorta Using Deformable Image Registration

Purpose: Accurate assessment of thoracic aortic aneurysm (TAA) growth is important for appropriate clinical management. Maximal aortic diameter is the primary metric that is used to assess growth, but it suffers from substantial measurement variability. A recently proposed technique, termed Vascular Deformation Mapping (VDM), is able to quantify three-dimensional aortic growth using clinical computed tomography angiography (CTA) data using an approach based on deformable image registration (DIR). However, the accuracy and robustness of VDM remains undefined given the lack of a ground truth from clinical CTA data, and furthermore the performance of VDM relative to standard manual diameter measurements is unknown. Methods: To evaluate the performance of the VDM pipeline for quantifying aortic growth we developed a novel and systematic evaluation process to generate 31 unique synthetic CTA growth phantoms with variable degrees and locations of aortic wall deformation. Aortic deformation was quantified using two metrics: Area Ratio (AR), defined as the ratio of surface area in triangular mesh elements, and the magnitude of deformation in the normal direction (DiN) relative to the aortic surface. Using these phantoms, we further investigated the effects on VDM’s measurement accuracy resulting from factors that influence quality of clinical CTA data such as respiratory translations, slice thickness and image noise. Lastly, we compare the measurement error of VDM TAA growth assessments against two expert raters performing standard diameter measurements of synthetic phantom images. Results: Across our population of 31 synthetic growth phantoms, the median ab- solute error was 0.048 (IQR: 0.077-0.037) for AR and 0.138mm (IQR: 0.227-0.107mm) for DiN. Median relative error was 1.9% for AR and < 6.4% for DiN at the highest tested noise level (CNR = 2.66). Error in VDM output increased with slice thickness, with highest median relative error of 1.4% for AR and 6.3% for DiN at slice thickness of 2.0 mm. Respiratory motion of the aorta resulted in maximal absolute error of 3% AR and 0.6 mm in DiN, but bulk translations in aortic position had a very small effect on measured AR and DiN values (relative errors < 1%). VDM-derived measurements of magnitude and location of maximal diameter change demonstrated significantly high accuracy and lower variability compared to two expert manual raters (p < 0.03 across all comparisons). Conclusions: VDM yields accurate, three-dimensional assessment of aortic growth in TAA patients and is robust to factors such as image noise, respiration-induced translations and differences in patient position. Further, VDM significantly outperformed two expert manual raters in assessing the magnitude and location of aortic growth despite optimized experimental measurement conditions. These results support validation of the VDM technique for accurate quantification of aortic growth in patients and highlight important several advantages over current measurement techniques.NIH R44HL145953http://deepblue.lib.umich.edu/bitstream/2027.42/166324/1/MedPhys-VDM-Synthetic-validation[1].pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/166324/3/VDM Vlidation Synthetic Phantom.pdfSEL

Fully Automated Pipeline for Measurement of the Thoracic Aorta Using Joint Segmentation and Localization Neural Network

Purpose: Diagnosis and surveillance of thoracic aortic aneurysm (TAA) involves measuring the aortic diameter at various locations along the length of the aorta, often using computed tomography angiography (CTA). Currently, measurements are performed by human raters using specialized software for 3D analysis, a time-consuming process, requiring 15-45 minutes of focused effort. Thus we aimed to develop a convolutional neural network (CNN)-based algorithm for fully automated and accurate aortic measurements. Approach: Using 212 CTA scans, we trained a CNN to perform segmentation and localization of key landmarks jointly. Segmentation mask and landmarks are subsequently used to obtain the centerline and cross-sectional diameters of the aorta. Subsequently, a cubic spline is fit to the aortic boundary at the sinuses of Valsalva to avoid errors related inclusions of coronary artery origins. Performance was evaluated on test set of 57 scans, with automated measurements compared against expert manual raters. Result: Joint training of segmentation and landmark localization tasks yielded higher accuracy for both tasks compared to networks trained for each task individually. Mean absolute error between human and automated was ≤ 1 mm at 6 of 9 standard clinical measurement locations. However, higher errors were noted in the aortic root and arch regions, ranging between 1.7 and 2.1 mm, although agreement of manual raters was also lower in these regions. Conclusion: Fully-automated aortic diameter measurements in TAA are feasible using a CNN-based algorithm. Automated measurements demonstrated low errors that are comparable in magnitude to those with manual raters, however, measurement error were highest in the aortic root and arch.http://deepblue.lib.umich.edu/bitstream/2027.42/177150/1/JMI.TAA.Automated.Seg.LLoc.pdfDescription of JMI.TAA.Automated.Seg.LLoc.pdf : Main Article. V1.7.2.23SEL

Vascular Deformation Mapping for CT Surveillance of Thoracic Aortic Aneurysm Growth

Background: Aortic diameter measurements in patients with thoracic aortic aneurysm (TAA) suffer from significant measurement variability and are unable to quantify aortic growth in a comprehensive, three-dimensional manner. Purpose: To develop and validate a technique for quantification of 3D growth based on deformable image registration in a cohort of patients with thoracic aortic aneurysm (TAA). Materials & Methods: We retrospectively identified a cohort of 50 patients with ascending and descending TAA with ≥2 computed tomography angiography (CTA) studies between 2006-2020; 12 patients were excluded yielding 38 patients (68 surveillance intervals) for analysis. 3D aortic growth was quantified using Vascular Deformation Mapping (VDM), a technique that uses deformable image registration to warp an aortic mesh constructed from baseline anatomy, with growth quantified as the ratio of change in surface area at each triangular mesh element (Area Ratio). Manual segmentations were performed by independent raters to assess inter-rater reproducibility. Registration error was assessed using manually placed landmarks. Agreement between VDM and diameter measurements was assess using Pearson’s correlation and Cohen’s kappa. Results: Average age was 69.0 ± 9.3 years, and the majority were female (n=21, 55%) with aneurysm of the ascending aorta (n=26, 69%). VDM was technically successful in 35/38 (89%) patients and 58/68 intervals (85%). Median registration error was 0.77 mm (IQR: 0.54-1.10 mm). Inter-rater agreement was high for aortic segmentation (Dice= 0.97 ± 0.02) and VDM-derived Area Ratio (bias= 0.0, limits of agreement: -0.03 to 0.03). There was strong agreement (r=0.85, p<0.001) between peak Area Ratio values and diameter change. VDM detected areas of growth outside of the maximally dilated segment in 6/14 (36%) patients with growth, none of which were detected by diameter assessment. Conclusion: VDM is a novel technique for reliable and comprehensive quantitative assessment of 3D aortic growth and growth patterns in TAA patients undergoing imaging surveillance.http://deepblue.lib.umich.edu/bitstream/2027.42/166323/1/VDM.Clinical.Validation.2.19.21.Draft.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/166323/3/VDM.Clinical.Validation.Submitted.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/166323/4/VDM.Clinical.Validation.FinalDraft.pdfhttp://deepblue.lib.umich.edu/bitstream/2027.42/166323/5/Burris et al.VDMinTAA.Radiology.2021.pdfSEL

Vascular Deformation Mapping of Abdominal Aortic Aneurysm

Abdominal aortic aneurysm (AAA) is a complex disease that requires regular imaging surveillance to monitor for aneurysm stability. Current imaging surveillance techniques use maximum diameter, often assessed by computed tomography angiography (CTA), to assess risk of rupture and determine candidacy for operative repair. However, maximum diameter measurements can be variable, do not reliably predict rupture risk and future AAA growth, and may be an oversimplification of complex AAA anatomy. Vascular deformation mapping (VDM) is a recently described technique that uses deformable image registration to quantify three-dimensional changes in aortic wall geometry, which has been previously used to quantify three-dimensional (3D) growth in thoracic aortic aneurysms, but the feasibility of the VDM technique for measuring 3D growth in AAA has not yet been studied. Seven patients with infra-renal AAAs were identified and VDM was used to identify three-dimensional maps of AAA growth. In the present study, we demonstrate that VDM is able to successfully identify and quantify 3D growth (and the lack thereof) in AAAs that is not apparent from maximum diameter. Furthermore, VDM can be used to quantify growth of the excluded aneurysm sac after endovascular aneurysm repair (EVAR). VDM may be a useful adjunct for surgical planning and appears to be a sensitive modality for detecting regional growth of AAAs